Experimental Design for the Evaluation of Medulator: an Integrated Online Case-based Learning Tool

Among its performance assessment capabilities, Medulator uniquely evaluates users’ Analogical Reasoning skills. We studied the effectiveness of Medulator’s Analogy Transfer Evaluation (ATE) and Case Summary components using various combinations of analogical case structures, distinguishing between cases’ surface characteristics (immediately apparent information such as patient age, gender, occupation, chief complaint, etc.) and structural characteristics (diagnosis determinants such as symptomatology, physical examination findings, test results, response to therapy, etc.). We measured 72 fourth-year medical students’ performance on different sequences of analogous cases (easy analogies followed by harder analogies and vice versa) under certain conditions. The control group performed a standard case assessment while the intervention group also did an ATE, which required the students to rate the structural similarities and differences between source cases (already completed, diagnosis known) and the target case (the current case, diagnosis unknown). We found that ATE alone had no effect on most dependent variables related to case performance, but a highly significant interaction (p=.009) was seen between ATE and case sequencing with respect to Correct Treatment scores. Specifically, when surface characteristics are not predictive of diagnosis, ATE is associated with improved Correct Treatment scores. Furthermore, a powerful effect (p<.002) of case summaries was seen with respect to Diagnostic Accuracy and Treatment Attempts. When students write their own case summaries, Diagnostic Accuracy scores increase and Treatment Attempts decrease. Case summaries improved the performance on more difficult cases (Bioterrorism) to a greater degree than less difficult cases (Cardiology). Additional time needed to write case summaries was not significant (p=.12). Finally, exit survey revealed a high degree of user satisfaction and perceived value of Medulator. Additional studies are currently being conducted which compare integrated Medulator cases to traditional classroom teaching at a major medical school. For this e-poster, study results will be posted for discussion and Medulator’s performance assessment capabilities will be demonstrated. Sample references: Gentner, D., & Holyoak, K. J. (1997). Reasoning and learning by analogy: Introduction. American Psychologist, 52, 32-34. Holyoak, K. J., & Thagard, P. (1997). The analogical mind. American Psychologist, 52, 35-44.